1. Experiments
   • Solution to confounding
   • What are the assumptions?
2. Trade-offs
   • interal validity
   • external validity
3. Conditioning

My wife just learned to will receive a COVID vaccination in the coming week

But, we know about the fundamental problem of causal inference and about confounding…

• How do we know that this vaccine actually reduces risk of infection?
• How do we know that the vaccine does not cause adverse reactions?

Clinical trials are experiments. Correlation between treatment and health outcomes are causal (do not have confounding) assuming that…

• assignment to treatment is random
• the only difference between treatment and control is the actual content of the vaccine. (blind placebo)

TO THE BOARD

Does the efficacy of vaccines in clinical trials translate to real world use??

POLL

• How might people who enroll in clinical trials be different.
• How might people behave differently in vaccine trials?

Might appear to be the only valid solution:

• if we don’t know how cases would behave counterfactually
• if we don’t know other causal factors affecting X and Y

Experiments, under weak assumptions, let us find an unbiased causal relationship between X and Y

• Experiments likely have “internal validity”

Internal Validity

A research design (choice of which cases to compare using correlation) has internal validity when the causal effect of X on Y it finds is not biased (systematically incorrect) / does not suffer from confounding.

• studies with strong internal validity imply that we have very good reason to believe that the correlation of X and Y we observe actually implies the causal effect of X on Y.
• because we can believe the assumptions (e.g. randomization)

What can we manipulate?

• economic growth? democracy? violence? hate speech?

Who/what cases can we study?

• who participates in psych labs? is it ethical to experiment on people in developing countries?

External Validity

is the degree to which the causal relationship we find in a study captures/is relevant to the causal relationship in our causal question/claim

• Study has external validity if the relationship found is true for the cases we are interested in

  • is study has sampling bias (sample in study different from population of interest), may lack external validity
  • E.g. Transgender canvassing

• Study has external validity if the causal variable in the study maps onto the concept/definition of the cause in the causal claim.

  • E.g. Fox News media effects

More internal validity (unbiased calculation of causal effect) comes at the cost in external validity (relevance of study sample or cause to the theory)

• easier to experimentally manipulate unimportant causal factors for small groups of people
• hard/unethical to experimentally manipulate important causal factors for society more broadly

You live in mid-19th century London.

• Every few years, hundreds to thousands of people are killed in cholera outbreaks
• To stop these deaths, you need to answer:

What causes the spread of cholera?

Dominant view was that “miasmas” or bad air caused diseases like cholera

• Wrong, yet it informed public health plans of the day

• John Snow, MD suggested cholera transmitted as “germ” in water.
• Leading doctors rejected Snow’s argument

Snow mapped cholera deaths of 1854 outbreak in SoHo.

• Broad Street Pump had fouled water (X)
• Proximity to the Broad Street Pump (water source) correlated with mortality (Y)
• Proximity to other pumps not related to mortality

Critics pointed out:

• Houses and sewers near Broad Street Pump built on 1665 plague burial site.
• Sewers produce foul odors from rotting material/human waste

Both might produce miasmas.

• maybe Miasmas \(\to\) Foul Water
• and Miasmas \(\to\) Cholera

Snow’s solution to confounding: compare people “near pump” w/ different water sources

Snow’s solution to confounding: compare people “far from pump” w/ different water sources

This solution to confounding is called…

conditioning

when we observe \(X\) and \(Y\) for multiple cases, we examine the correlation of \(X\) and \(Y\) within groups of cases that are the same on confounding variables \(W, etc. \ldots\)

How does conditioning solve the problem?

• Cases compared have same values on confounding variable \(W\)
• In these groups, \(W\) cannot affect \(X\) or \(Y\)
• “Backdoor” path from \(X\) to \(Y\) is “blocked”

Compare COVID infections between vaccinated and un-vaccinated, identical on…

• Postal Code
• age, sex, race, ethnicity
• Number of previous COVID tests

Confounding:

• What are the assumptions?